High-quality, reduced data rate streaming video production and monitoring system

ABSTRACT

A multi-format digital video production system is capable of maintaining full-bandwidth resolution of subject material while providing professional quality editing and manipulation of images intended for digital television and other applications, including digital HDTV programs and specialized video monitoring applications. This allows emerging broadband video transmission media, including Internet broadcast schemes, to overcome existing technology limitations. The approach facilitates high-quality/large-screen video production and monitoring through the use of conventional broadband channels, including those which currently only exhibit bandwidths on the order of 4 Mbps. In formats utilizing substantially 24 fps progressive scan multi-format system, direct streaming is made possible from HDTV (16:9) high-quality data, thereby expanding market applications which require these higher levels of resolution, bits per pixel, and so forth.

REFERENCE TO RELATED APPLICATION

This application claims priority from U.S. patent application Ser. No.10/664,244, filed 17 Sep. 2003, which claims priority from U.S.Provisional Patent Application Ser. No. 60/411,474, filed 17 Sep. 2002,the entire content of both which is incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates generally to digital video and, moreparticularly, to a multi-format digital video production system capableof maintaining full-bandwidth resolution while providing professionalquality editing and manipulation of images for various applications,including digital HDTV and specialized video monitoring.

BACKGROUND OF THE INVENTION

Traditional systems for video production either rely on uncompressedvideo signals (for example, SMPTE 4:4:4 or 4:2:2), standard compressedMPEG-2 4:2:2P@ML signals, or other signals that have undergone onlyminimal compression, such as the (approximately) 5:1 compressionutilized for DVCPRO and DVCAM equipment by Panasonic and Sony. However,the bandwidth required for these high-quality signals still is too greatfor many broadcast and industrial applications, particularly those thatrequire the level of detail available in HDTV images.

Due to the high-bandwidth demands of high-quality signals, typicaldistribution systems utilize only the highest quality levels for thehead-end equipment and the first part of the signal distribution chain.Furthermore, because of network traffic due to multiple users (as forexample, in a cable television distribution system), the last leg of thesignal path utilizes a more highly compressed signal, to maximize theusage of the available bandwidth. In most cases, this requires that theoriginal signal be decompressed, and then re-compressed at a much highercompression ratio, so that less bandwidth is required for the finalportion of the path.

FIG. 1 is a diagram which illustrates the way in which conventionalbroadband transmission media are used. Progressive-scan devices areindicated at 102, and include 35 mm film 106, 24 frame-per-second (fps)cameras 108, and the inventive 24P camera system 110 described infurther detail herein. Film production and television production areindicated with the vertical box 112, and Internet/broadband applicationsare shown at 120. Interlace scan devices 114 include 30 fps NTSC 116 and25 fps PAL 118. Although suitable for certain film and televisionproduction applications, interlaced video 114, whether NTSC 116 or PAL118, is inferior for Internet and broadband applications 120, since thedelivered video quality is less than that possible with progressivedisplay, regardless of compression. Even using a progressive format,however, film (35-mm) 106 and high-end 24 fps progressive camera inputs108 are deficient in terms of quality, due to the need for high levelsof compression later in the signal transmission path.

Accordingly, the need remains for an approach to video production andmonitoring which allows the levels of quality that users have come toexpect at their receiving terminals, while utilizing existing broadbandmedia and other conventional technologies to optimize the signalstorage, processing, and transmission path performance.

SUMMARY OF THE INVENTION

This invention resides in a multi-format digital video production systemcapable of maintaining the full-bandwidth resolution of the subjectmaterial, while providing professional quality editing and manipulationof images intended for digital television and for other applications,including digital HDTV programs and specialized video monitoringapplications.

Broadly, this invention allows emerging broadband video transmissionmedia, including Internet broadcast schemes, to overcome existingtechnology limitations. In the preferred embodiment, for example, theapproach facilitates high-quality/large-screen video production andmonitoring through the use of conventional broadband channels, includingthose which currently only exhibit bandwidths on the order of 4 Mbps. Inmore specific examples, in formats utilizing a 24 fps progressive scanmulti-format system, direct streaming is made possible from HDTV (16:9)high-quality data, thereby expanding market applications which requirethese higher levels of resolution, bits per pixel, and so forth.

This system, now known as the “Direct Stream Cinema System,” is based onoptimizing the entire signal path, utilizing 4:2:2 color processing andbit rates typically in the range of 2-6 Mbps. It begins with digitizingand compressing the output of the optical pickup and graphics processor(including appropriate processing, such as noise reduction andresolution enhancement) and carries through the processing circuitry tothe receiving terminal device at the user end of the transmission chain.Signal quality is preserved throughout the process, by eliminating theneed to decompress a lower-compression signal from a camera, videorecorder, or other source device for editing or other purposes, and thenre-compressing the signal at a much higher rate for transmissionpurposes.

A high-quality, reduced-data-rate digital video system according to apreferred embodiment includes a source of a streaming video programhaving a progressive-scanned image with a frame rate of less thansubstantially 24 fps; a video server in communication with the sourcefor storing the program; and one or more computers in networkcommunication with the video server for locally displaying the programor portions thereof.

In a “direct stream” implementation the locally displayed program orportions thereof are in the same format as the streaming video programreceived form the source. The system and method may further include apersonal-computer-based control of the camera/input device, monitor forthe streaming video program received from the source, or other PC-basedcapabilities. The streaming video program may be received through anetwork connection, and the video server includes one or more of thefollowing for storing the program: a micro-disk, portable HDD,memory-stick, optical storage, or magneto-optical storage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram which illustrates the way in which conventionalbroadband transmission media are used, showing how high compression andinterlaced video are poor choices for higher quality applications;

FIG. 2 is a diagram which shows the concepts behind the various versionsof the “Direct Stream Cinema” systems;

FIG. 3 is a diagram which illustrates a particular configurationconstructed in accordance with the invention, namely, a videosurveillance system;

FIG. 4 is a drawing which shows a different particular layout accordingto the invention, in this a streaming production system;

FIG. 5 is a diagram which shows the way in which the instant inventionimplements HDTV quality video at a very low overall system cost; and

FIG. 6 is a diagram which shows the quality levels provided by theconsumer-type implementation of the “Direct Stream Cinema” systems.

DETAILED DESCRIPTION OF THE INVENTION

This invention overcomes the limitations of the existing art byproviding a video production/monitoring capability capable oftransmitting HDTV (16:9) quality video utilizing existing broadbandbandwidths of [such as 4 Mbps (1024×576 pixels) or greater], therebyovercoming the traditional problem of conserving bandwidth whilepreserving quality.

The “Direct Stream Cinema System” preferably utilizes a 24 fpsprogressive camera format which, through the use of proprietarymulti-format production techniques (110), facilitates Internet andbroadband applications, including streaming services 122, Internet TV,video monitoring/security 124, and 35 mm/HDTV/DVD output capabilities126. The approach does not require an HDTV quality video camera orrecording, however, but nevertheless facilitates HDTV quality, directvideo monitoring, off-line editing, and other capabilities at a greatreduction in total system cost.

With respect to streaming applications, the video data may betransmitted directly to a central server through a network environment,resulting in both a comparatively small capacity storage requirement andalso in other advantages over existing approaches. In one disclosedexample, HDTV quality video with an aspect ratio of 16:9 is achieved,having a horizontal resolution of 1024×576, with the potential forup-conversion to 1920×1080. This resolution, equivalent to a 42-inchplasma display, is accomplished with a data rate of 4 Mbps, more orless, enabling recording to occur at 2 GBytes/hr, whereas current HDTVrequires more than 100 GBytes/hr. Various video formats are possiblethrough the use of proprietary multi-format progressive systems andframe rates, which may vary up to 24 fps (or greater) in the preferredembodiment.

Newer media players, such as Microsoft's new “Corona” technology, whichis scheduled to be released with the latest version of the Windows MediaPlayer (Series 9), are aimed at signal distribution systems utilizing adata rate of 6 Mbps, using MPEG 4 and other comparable compressiontechniques. However, such technology also provides for bit rates in therange of 2-4 Mbps, being directed towards applications such asarchiving, streaming video, and off-line viewing. At these data rates,it is possible to store 100 hours of video in only 180 GB of storage[(100 hr)×(3600 sec/hr)×(4 Mbps)/(8 b/B)].

FIG. 2 discloses three of the many potential implementations of the“Direct Stream Cinema” system: Professional cameras and Camcorders 210,Consumer Camcorders 212, and Digital-Still-Camera recorders 214.According to the invention, the entire process 202 may use digitalcomponent (4:2:2) processing, preferably based upon a 3-CCD 24P input204, through graphic processing and compression at 206, to storage 208,whether on a hard drive, digital video disk, memory card, or othermedium. Video stored in this manner is suitable for on-line editingapplications, using PC plug-in hardware cards from companies like Matrox(Perphelia) and ATI (Radion 9200/9800), Nvidia (GeForce FX). However,these conventional off-the-shelf-types of cards require modification, sothat they would be equipped with true DV or SDI digital video outputs,thereby providing compatibility with both HDTV and standard NTSCformats, including analog, Y-C component formats, and composite videooutputs. In addition, software packages such as Adobe Premier 6.5, andUlead MediaStudio 7, when utilized with a high-end PC (3 GHz or higherprocessing speed), are capable of providing sophisticated editingcapabilities.

The resulting signal can be stored, in an AVI format, for example, on ahard disk drive. Currently, these PC cards only are being used for SDTV,but in the future, they will be capable of HDTV recording, and forspecialized industrial applications; for HDTV applications, a newdecoder board would be used.

The preferred storage and distribution format according to the inventionis 1024×576@24 fps. Compression ratios of 100:1 are practical for SDTV,and 400:1 for HDTV. In addition, the system is scalable, for example, tothe following:

200 Kbps@1 fps

500 Kbps@3 fps

1 Mbps@6 fps

2 Mbps@12 fps

4 Mbps@24 fps

Comparisons of the output quality of a variety of PC-video display cardsutilizing both interlaced and progressive signals and alsoframe-rate/standards-conversion indicate a need to optimize the signalprocessing. For conversions from interlaced PAL signals to NTSC, thesecards produce outputs with noticeable frame skipping and jumping.However, from a progressive PAL signal (i.e., greater than 50 fpsprogressive), the severity of artifacts is greatly reduced. Newer PCgraphics cards produce significantly better results, which suggests thatthey may have adopted the frame-rate conversion techniques disclosed inU.S. Pat. No. 5,999,220, entitled “Multi-Format Audio/Video ProductionSystem with Frame Rate Conversion” and U.S. Pat. No. 6,370,198B1,entitled “Wide-band Multi-Format Audio/Video Production System withFrame Rate Conversion,” the entire content of both being incorporatedherein by reference.

In preferred embodiments, signals at the head-end of a signaldistribution system are converted to progressively scanned signals. Aframe rate of 24 fps preferably is employed, in order to optimize theutilization of the available bandwidth. In the next step, the signalsare compressed to create a data stream at 2-4 Mbps (for 1024×576@24 fps)or 4-6 Mbps (for 1280×720@24 fps. These signals may be stored forsubsequent transmission to receiving terminal equipment (such as PCs,cable boxes, personal video recorders, display monitors, or otherterminal equipment), or immediately transmitted over a signaldistribution system, which may be wired, wireless, satellite, or othermedium, including physical media such as CD-ROMs, DVDs, etc.). Thisreceiving terminal equipment may be located at multiple remote sites,may be located at multiple sites within a single facility, or may beconfigured as a combination of local and remote sites.

In an alternative embodiment, signals may be received from multiplesources, including one or more remote sources, and are collected at acentral location for viewing, storage, or both. The signals preferablyare transmitted to the central site as compressed, progressively-scannedstreaming video signals, employing data rates in the range of 2-4 Mbps.As in other embodiments, 24 fps is preferably used, although the framerate may be greater or less, may be variable or fixed, and may bemodified under control of a local operator, or may be modifiedautomatically in response to a predetermined set of criteria, utilizingsensors at the physical location of the camera or signal source, or viaremote control from a central site, either under control of an operator,or automatically in response to a predetermined set of criteria. Thesource signal frame rate and image size may be different for each sourcesignal, and the frame rate and image size of a source signal in theformat stored need not be identical to the frame rate and image size inthe format displayed.

Currently, ½-inch 3-CCD cameras are available for less than $10,000, and⅓-inch 3-CCD cameras are available for approximately $5,000. As such, itis already practical and economical to implement this type of system fora range of commercial/industrial applications, for example:

Airport security

Monitoring of remote natural areas, such as forests

Auto crash testing

Public building (Court, Government office, School) security

Hospital security

Educational/instructional

FIG. 3 is a diagram which illustrates a particular configurationconstructed in accordance with the invention, namely, a videosurveillance system. In this case, the signals from multiple cameras 302are transmitted as streaming sources at relatively low data rates, onthe order of 200K to 4 Mbps, with 1 to 24 fps variable frame rates viabroadband connection 310. As discussed above, this conserves videoserver 312 storage requirements, facilitating one hour of storageutilizing only 2 GB of capacity. This information may then benetwork-accessed by one or more monitoring control systems 314,preferably using multi-screen displays, and optionally including alarmsor other features using graphic analysis or other methodologies.

The advantages of this approach are many, in addition to the ability touse existing broadband infrastructures supporting data transfers in therange 1:4 Mbps, the systems may be built at 1/10th cost of conventionalHDTV systems. High-quality monitoring is capable, as is direct networkconnectivity. The use of a generic PC-based server can easily handle alarge monitoring application. The resulting configuration improvessecurity, at banks, for example, while reducing mistakes due to humanerror. Operating efficiency is improved for medical applications, forexample, along with reliability and monitoring efficiency (speed).Overall, the system is physically compact.

FIG. 4 is a drawing which shows a different particular layout accordingto the invention, in this case a streaming production system which maybe implemented with Professional-quality equipment. Again, a camera 402producing HDTV quality video transmits at a relatively low data rate asa streaming source to a program editing facility 410 through a directconnection 412, enabling various operator controls including, but notlimited to, frame-by-frame control, variable playback, forward/reverse(bi-directional) playback, and so forth. A decision list is generated ona scene-by-scene basis, with AVI file conversion being used forcompatibility with PC non-linear editing. Alternative formats wouldinclude, for example, MPEG-4, Windows Media 9, or Divx (which even canbe edited, utilizing one of the available software packages for editing.The source material and EDL (Edit Decision List) codes are stored in astreaming server, with the resulting modest requirements facilitating anhour of storage within a Gigabyte of memory (for SDTV at 2 Mbps) orwithin two Gigabytes of memory (for HDTV at 4 Mbps. The streaming videois output to one or more likely multiple viewing stations, utilizing aneven lower data rate of, perhaps, less than two Mbps. Conventional SDTVsignals utilizing a compressed DV-type output typically would beprovided at 25-50 Mbps. HDTV-type signals utilizing a compressedSDI-type output would be provided at 100-300 Mbps; however, the signalmanipulations within the system and before the output stages wouldutilize the more efficient and compact 4 Mbps files and signal streams.

This system application offers numerous features and advantages over atraditional system, which requires a more traditional recording andediting system 406, and which does not allow a direct connection viapath 408. Using the approach described above, results in a dramaticreduction and system cost (under $10.000 vs. $100.000 or more at currentprices). Full digital component processing (4:2:2) is achieved without aloss in quality, and excessive hard disk drives are not required forediting; rather, a generic PC is capable of editing the program (10gigabytes vs. terabytes for traditional HDTV). The advantages includes areduced HDTV production cost and time without a separate data capturestep. The invention is not limited in term so video format or streaming,as all existing and yet to be developed formats may be accommodated.

FIG. 5 is a diagram which shows the way in which the instant inventionimplements HDTV quality video at a very low overall system cost. At thehigh end, an HDTV camera with a format 502 of 16:9 at 1920×1080 pixelsuses some 2 million pixels per image as the source, which is reduced at504 to less than 1 Megapixels or thereabouts due to interlace losses,bandwidth limiting, compression losses and so forth, resulting in anactual resolution of 70 percent of the original. Even so, equipmentexhibiting this level of performance currently involves hardware costsof approximately $200,000.

While broadcast quality video 508 (standard definition at 4:3) costsmuch less, the image quality is reduced dramatically, to a frame size of720×480 pixels (4:3, 30 fps). According to the invention, however,utilizing a 24 fps scan and proprietary multi-format system at 506, a24P image at 1024×576 or 1280×720 can be generated having an aspectratio of 16:9, exhibiting a quality comparable to conventional HDTVbroadcast, but at a cost of under $10,000. A typical surveillance image,at 320×240 and <15 fps is shown at 510 for comparison purposes.

For any of these implementations (Professional, Camcorder, Surveillance,or Consumer), a key part of the system resides in the optimization ofthe entire processing scheme, with an eye towards the end-user qualitylevel. For example, in the case of modem plasma-display units, thecapability of the individual unit largely is determined by the physicaldimensions of the screen: 32″ displays are supplied as capable of848×477 pixels; 42″ displays are supplied as capable of 1024×576 pixels;50″ displays are supplied as capable of 1280×720 pixels. Becausemultiple tests have demonstrated that “film quality” as measured at thetheatrical projection screen only provides approximately 700 lines ofresolution (see, for example, A. Kaiser, H. W. Mahler, and R. H. McMann,SMPTE Journal, June, 1985), 1024×576, or at most 1280×720, provides theoptimum display quality; 1920×1080 or other higher-pixel-count systemsare not required.

Another key feature of the system is the utilization of compressiontechnology. Most origination-quality systems rely on intra-framecompression (such as Motion-JPEG), which is limited to 3:1 or 4:1 forthis type of application. Further downstream in the processing andtransmission chain, much higher inter-frame-based compression ratios areneeded (such as MPEG-2), in order to make signal distribution practicaland economical. The instant invention contemplates high compressionratios throughout the process, achieving in excess of 100:1 compression.In this way, the use of “intermediate” formats, such as DVC-PRO orDV-CAM no longer are required. Furthermore, the reduced data ratesrequired for the system eliminates the need for extremely large capacityhard-disk recording capability, enabling editing on most of today'sconventional PCs.

However, in order to achieve these kinds of compression ratios withoutsacrificing quality, the preferred embodiment employs 24 fps signals(which, evidently, saves 20% of the data rate required for a 30 fpssignal), and also progressive-scanning (which is over 50% more efficientthan compression of interlaced signals). Many compression schemes areavailable, including, for example, industry standards such as MPEG-4,and proprietary systems such as Microsoft Windows Media 9, Divx, andWavelet-type compression. The resulting data rates easily are conveyedover conventional distribution paths, such as satellite, cable, andbroadcast systems, requiring only 1-2 Mbps for SDTV-type signals, and 6Mbps for HDTV-type signals.

As shown in FIG. 6, in Consumer-type applications, it is common toemploy digital still camera systems, utilizing high-speed shutters toprovide video program sourcing. For example, at a resolution of 320×240and <15 fps (4:3) the results are limited to relatively low-qualityrecordings for relatively limited recording times. In addition, manyartifacts are imparted to the recordings, such as motion artifacts andpicture hesitation or jumps. Photo jpeg compression does not reproducesmooth motion, recording time is limited, and audio quality is poor.

However, consumer cameras are producing increasingly high qualityrecording. despite their small size and low cost. By employing thetechniques disclosed herein, DV-quality recordings for more than onehour are practical, and S-VHS-quality recordings for more than two hourscan be achieved. In addition, video editing is simplified, as no step ofcapturing to the PC is required—editing can proceed directly from cameramemory cards or other storage devices (including hard-disk, opticaldisc, DVD, etc.), and the quality is preserved throughout the process.In addition, the resulting recordings are compatible with variousstreaming conventions, such as those supported by Microsoft and RealNetworks video. This same system of video processing without a step ofcapturing the signal to the PC applies equally as well to Professionaland Camcorder applications.

The reader will appreciate that the practical application of the instantinvention has significant implications in many fields. For example,Digital Asset Management systems typically employ highly-compressed“proxies” to convey the content of much less-compressed primary programmaterials, thereby enabling Edit Decision Lists to be developed from the“proxies” and then used to edit the final program using the primaryprogram material. With the much more efficient signal processing methodsprovided herein, it is not necessary to create the separate proxies, asthe primary signals themselves are provided at much lower data ratesthan traditionally have been available for these materials, making themsuitable for use in a single-step on-line editing application.

The “Direct Stream Cinema System” is based on optimizing the entiresignal path, utilizing 4:2:2 color processing and bit rates typically inthe range of 1-2 Mbps for SDTV-quality video and 4-6 Mbps forHDTV-quality video. It begins with digitizing and compressing the outputof the optical pickup and graphics processor (including appropriateprocessing, such as noise reduction and resolution enhancement), so thatfrom the onset the data rate is set and then maintained through theinternal processing circuitry, recording steps, and through thedistribution steps to the receiving terminal device at the user end ofthe transmission chain. Signal quality is preserved throughout theprocess, by eliminating the need to decompress a lower-compressionsignal from a camera, video recorder, or other source device for editingor other purposes, and then re-compressing the signal at a much higherrate for transmission purposes. Thus, there is no distinct“intermediate” format of any kind, as the original video format obtainedfrom the optical pickup or other source device is maintained through theentire path to the receiving terminal device.

Note that, to a certain extent, the resolution sizes and pixels, as wellas the prices, and other data are associated with current technology,and are anticipated to vary in time as technology improves and matures.Nevertheless, the inventive approach of applicant will at all timesresult in a substantial decrease in system cost while preserving thehighest possible quality, even at limited bandwidths. Additionally, inall embodiments of the invention, techniques such as pixel interpolationmay advantageously be used to further enhance image resolution/quality.

We claim:
 1. A method of viewing, on a remote viewing device of a videosurveillance system, multiple simultaneously displayed and stored videoimages, comprising the steps of: receiving video images at a personalcomputer based system from a plurality of video sources, wherein each ofthe plurality of video sources comprises a camera of the videosurveillance system; digitizing any of the images not already in digitalform using an analog-to-digital converter; displaying one or more of thedigitized images in separate windows on a personal computer baseddisplay device, using a first set of temporal and spatial parametersassociated with each image in each window; converting one or more of thevideo source images into a selected video format in a particularresolution, using a second set of temporal and spatial parametersassociated with each image; contemporaneously storing at least a subsetof the converted images in a storage device in a network environment;providing a communications link to allow an external viewing device toaccess the storage device; receiving, from a remote viewing deviceremoted located remotely from the video surveillance system, a requestto receive one or more specific streams of the video images;transmitting, either directly from one or more of the plurality of videosources or from the storage device over the communication link to theremote viewing device, and in the selected video format in theparticular resolution, the selected video format being a progressivevideo format which has a frame rate of less than substantially 24 framesper second using a third set of temporal and spatial parametersassociated with each image, a version or versions of one or more of thevideo images to the remote viewing device, wherein the communicationlink traverses an external broadband connection between the remotecomputing device and the network environment; and displaying only theone or more requested specific streams of the video images on the remotecomputing device.
 2. The method of claim 1, wherein the remote viewingof the given video stream is contemporaneous with the live viewing andstorage.
 3. The method of claim 1, wherein the remote viewing of thegiven video stream is conducted subsequent to the live viewing andstorage.
 4. The method of claim 1, wherein the remote computing deviceand the network environment communicate through the Internet.
 5. Themethod of claim 1, wherein the broadband connection has a bandwidth of2-6 Megabits per second (Mbps).
 6. A method of viewing, on a remoteviewing device of a video surveillance system, multiple simultaneouslydisplayed and stored video images, comprising the steps of: receivingvideo images at a personal computer based system from a plurality ofvideo sources, wherein each of the plurality of video sources comprisesa camera of the video surveillance system; digitizing any of the imagesnot already in digital form using an analog-to-digital converter;displaying one or more of the digitized images in separate windows on apersonal computer based display device, using a first set of temporaland spatial parameters associated with each image in each window;converting one or more of the video source images into a selected videoformat in a particular resolution, using a second set of temporal andspatial parameters associated with each image; contemporaneously storingat least a subset of the converted images in a storage device in anetwork environment; providing a communications link to allow anexternal viewing device to access the storage device; receiving, from aremote viewing device remoted located remotely from the videosurveillance system, a request to receive one or more specific streamsof the video images; transmitting, either directly from one or more ofthe plurality of video sources or from the storage device over thecommunication link traversing the Internet to the remote viewing device,and in the selected video format in the particular resolution, theselected video format being a progressive video format which has a framerate of less than substantially 24 frames per second using a third setof temporal and spatial parameters associated with each image, a versionor versions of one or more of the video images to the remote viewingdevice; and displaying only the one or more requested specific streamsof the video images on the remote computing device.